Abstract

OBJECTIVES: Equitable involvement of patients and clinicians in setting research and funding priorities is ethically desirable and can improve the quality, relevance and implementation of research. Survey methods used in previous priority setting projects to gather treatment uncertainties may not be sufficient to facilitate responses from patients and their lay carers for some health care topics. We aimed to develop a new model to engage patients and clinicians in setting research priorities relating to life after stroke, and to explore the use of this model within a James Lind Alliance (JLA) priority setting project. METHODS: We developed a model to facilitate involvement through targeted engagement and assisted involvement (FREE TEA model). We implemented both standard surveys and the FREE TEA model to gather research priorities (treatment uncertainties) from people affected by stroke living in Scotland. We explored and configured the number of treatment uncertainties elicited from different groups by the two approaches. RESULTS: We gathered 516 treatment uncertainties from stroke survivors, carers and health professionals. We achieved approximately equal numbers of contributions; 281 (54%) from stroke survivors/carers; 235 (46%) from health professionals. For stroke survivors and carers, 98 (35%) treatment uncertainties were elicited from the standard survey and 183 (65%) at FREE TEA face-to-face visits. This contrasted with the health professionals for whom 198 (84%) were elicited from the standard survey and only 37 (16%) from FREE TEA visits. CONCLUSIONS: The FREE TEA model has implications for future priority setting projects and user-involvement relating to populations of people with complex health needs. Our results imply that reliance on standard surveys may result in poor and unrepresentative involvement of patients, thereby favouring the views of health professionals.
Hide

Abstract

BACKGROUND: Despite a strong movement towards active patient involvement in healthcare policy decisions, systematic and explicit consideration of evidence of this research on patient preferences seems limited. Furthermore, little is known about the opinions of several stakeholders towards consideration of research evidence on patient preferences in healthcare policy decisions. This paper describes the protocol for an explorative study on the integration of research on patient preferences in healthcare policy decisions. The study questions: to what extent research evidence on patient preferences is considered in current procedures for healthcare policy decisions; opinions of stakeholders regarding the integration of this type of evidence in healthcare policy decisions; and what could be a decision framework for the integration of such research evidence in healthcare policy decisions. Methods/design: The study is divided in three sub-studies, predominantly using qualitative methods. The first sub-study is a scoping review in five European countries to investigate whether and how results of research on patient preferences are considered in current procedures for coverage decisions and clinical practice guideline development. The second sub-study is a qualitative study to explore the opinions of stakeholders with regard to the possibilities for integrating evidence on patient preferences in the process of healthcare decision-making in the Netherlands. The third sub-study is the development of a decision framework for research on patient preferences. The framework will consist of: a process description regarding the place of evidence on patient preferences in the decision-making process; and a taxonomy describing different terminologies and conceptualisations of 'preferences'and an overview of existing methodologies for investigating preferences. The concept framework will be presented to and discussed with experts. DISCUSSION: This study will create awareness regarding the existence and potential value of research evidence on patient preferences for healthcare policy decision-making and provides insight in the methods for investigating patient preferences and the barriers and facilitators for integration of such research in healthcare policy decisions. Results of the study will be useful for researchers, clinical practice guideline developers, healthcare policy makers, and patient representatives.
Hide

Abstract

BACKGROUND: There is an increasing international interest in patient and public involvement (PPI) in research, yet relatively little robust evidence exists about its impact on health and social care research. OBJECTIVE: To identify the impact of patient and public involvement on health and social care research. DESIGN: A systematic search of electronic databases and health libraries was undertaken from 1995 to 2009. Data were extracted and quality assessed utilizing the guidelines of the NHS Centre for Reviews and Dissemination 2009 and the Critical Appraisal Skills Programme (CASP). Grey literature was assessed using the Dixon-Woods et al. (2005) checklist. INCLUSION CRITERIA: All study types that reported the impact PPI had on the health and/or social care research study. MAIN RESULTS: A total of 66 studies reporting the impact of PPI on health and social care research were included. The positive impacts identified enhanced the quality and appropriateness of research. Impacts were reported for all stages of research, including the development of user-focused research objectives, development of user-relevant research questions, development of user-friendly information, questionnaires and interview schedules, more appropriate recruitment strategies for studies, consumer-focused interpretation of data and enhanced implementation and dissemination of study results. Some challenging impacts were also identified. CONCLUSION: This study provides the first international evidence of PPI impact that has emerged at all key stages of the research process. However, much of the evidence base concerning impact remains weak and needs significant enhancement in the next decade.
Hide

Abstract

Having patients, doctors, health plan managers, hospital executives, and other stakeholders participate in the design of comparative effectiveness studies can ensure that this vital research focuses on the evidence gaps most relevant to health care decision makers. Through a qualitative assessment of case studies, we identify five key principles for the effective engagement of a broad coalition of participants in research intended to improve health care and control costs. Those principles are to ensure balance among the participating stakeholders; get participants to "buy in" to the process and understand their roles; provide neutral and expert facilitators for research discussions; establish connections among the participants; and keep the participants engaged throughout the research process.
Hide

Abstract

BACKGROUND:Recognising the importance of increased patient participation in healthcare decisions leads decision makers to consider effective ways to incorporate patient perspectives in Health Technology Assessment (HTA) processes. The implementation of local health HTA units in university hospitals in Quebec provides a unique opportunity to foster an increased participation of patients in decisions regarding health technologies and clinical interventions. This project explores strategies that could be effective in involving patients in HTA activities at the local level. To do so, three objectives are pursued: 1) To synthesise international knowledge and experiences on patient and public involvement in HTA activities; 2) To explore the perceptions of stakeholders (administrators, clinical managers, healthcare professionals, HTA producers, and patients) regarding strategies for involving patients in various HTA activities; and 3) To produce a consensual strategic framework that could guide interventions for involving patients in HTA activities at the local level. METHODS:A systematic review of the literature will be conducted to synthesise international knowledge and experiments regarding the implication of patients and public in HTA. Then, focus groups will be carried out with representatives of various stakeholder groups in order to explore their perceptions regarding patient participation in HTA. Based on findings from the systematic review and the focus groups, a framework to support patient participation in HTA activities will be proposed. It will then be validated during a deliberative meeting with the research team, composed of scientists and decision makers, and representatives from different groups involved in HTA in Quebec. This deliberative meeting will aim at identifying the type and the degree of participation as well as the adequate timing for involving patients in local HTA activities. DISCUSSION:Given the actual state of evidence, integrating patient perspective in HTA activities has the potential to improve the quality of healthcare services. This study provides an opportunity to bridge the gap between HTA producers and its ultimate end-user: the patient. It will provide guidance to support local HTA units in Quebec and elsewhere in their decisions regarding patient participation. The framework developed could be applied to design and implement strategies for involving patients in HTA activities.
Hide

Abstract

OBJECTIVES: To look at the processes and outcomes of identification and prioritisation in both national and regional R&D programmes in health and elsewhere, drawing on experiences of success and failure. Also to identify the barriers to, and facilitators of, meaningful participation by consumers in research identification and prioritisation. DATA SOURCES: Electronic databases and interviews with UK consumers and research programme managers. REVIEW METHODS: A framework was devised for examining the diverse ways of involving consumers in research. It identified key distinguishing features as: the types of consumers involved; whether consumers or researchers initiated the involvement; the degree of consumer involvement (consultation, collaboration or consumer control); forums for communication (e.g. committees, surveys, focus groups); methods for decision-making; and the practicalities for implementation. Context (institutional, geographical and historical setting) and underpinning theories were considered as important variables for analysing examples of consumer involvement. This innovative framework was then applied to the review data from reports selected for inclusion and interviews. RESULTS: The study found 286 documents explicitly mentioning consumer involvement in identifying or prioritising research topics. Of these, 91 were general discussions, some of which included a theoretical analysis or a critique of research agendas from a consumer perspective, 160 reported specific efforts to include consumers in identifying or prioritising research topics and a further 51 reported consumers identifying or prioritising research topics in the course of other work. Detailed reports of 87 specific examples were identified. Most of this literature was descriptive reports by researchers who were key actors in involving consumers. A few reports were written by consumer participants. Fewer still were by independent researchers. Our conclusions are therefore not based on rigorous research, but implications for policy are drawn from individual reports and comparative analyses. CONCLUSIONS: Productive methods for involving consumers require appropriate skills, resources and time to develop and follow appropriate working practices. The more that consumers are involved in determining how this is to be done, the more research programmes will learn from consumers and about how to work with them. Further success might be expected if research programmes embarking on collaborations approach well-networked consumers and provide them with information, resources and support to empower them in key roles for consulting their peers and prioritising topics. To be worthwhile, consultations should engage consumer groups directly and repeatedly in facilitated debate; when discussing health services research, more resources and time are required if consumers are drawn from groups whose main focus of interest is not health. These barriers can largely be overcome with good leadership, purposeful outreach to consumers, investing time and effort in good communication, training and support and thereby building good working relationships and building on experience. Organised consumer groups capable of identifying research priorities also need to find ways of introducing their ideas into research programmes. Further research is suggested to develop and evaluate different training methods, information and education and other support for consumers and those wishing to involve them; to address the barriers to consumers'ideas influencing research agendas; and to carry out prospective comparative studies of different methods for involving consumers. Research about collective decision-making would also be further advanced by addressing the processes and outcomes of consensus development that involves consumers.
Hide

Abstract

The involvement of consumers in health research is now Department of Health policy within the UK. Despite the existence of policy directives, there is a dearth of knowledge on the effects of such involvement. This paper critically reviews the state of our knowledge on this issue, and maps out a research agenda with the aim of stimulating systematic, empirical inquiry into consumer involvement in health research. The paper discusses definitions of 'the consumer'; considers why consumer involvement is believed to be important to health research; traces the development of the policy; analyses the epistemological and methodological implications of the policy; discusses the various levels of consumer involvement in research; and outlines the objections to the policy that have been put forward by clinicians and researchers. Four questions were identified during the review as being in need of theoretical and empirical attention: (1) how can consumer involvement in health research be further conceptualised? (2) how and why does consumer involvement influence health research? (3) how can the influence of consumers in health research be measured and evaluated? and (4) what factors are associated with 'successful'consumer involvement in health research?
Hide

Abstract

BACKGROUND: Health-care organizations need to prioritize their resource use and should incorporate the public's preferences into their priority setting process. METHODS: We apply a discrete choice experiment (DCE) to obtain weights, from the public, for use in a priority setting exercise. Ten attributes were chosen: location of care, public consultation, use of technology, service availability, patient involvement, management of care, evidence of effectiveness, health gain, risk avoidance and priority area. From the DCE responses, weighted benefit scores were calculated and used to rank development bids from across a health-care organization. RESULTS: Sixty-eight members of the public completed the DCE. All attributes except risk avoidance were significant. The most important attribute levels were a large health gain to many people: care being provided in teams, using latest or cutting-edge technology and 24 h service availability. Local priorities were valued higher than national priorities. Ninety-five bids were ranked in order of overall score. The ranked list of development bids provided a useful tool to inform prioritization decisions. CONCLUSIONS: DCEs can offer a theoretically valid and practical means of incorporating the views of the public in an accessible, transparent and streamlined decision-making process when health-care organizations are prioritizing their resources.
Hide

Abstract

Decision makers are pressed to involve the public in priority setting. However, public input is only one form of evidence. So, how can information from the public be combined with other knowledge? The authors qualitatively analysed articles that explicitly address this question. We identified the other forms of information that tend to be used in conjunction with public input, the degree to which members of the public are asked to be the integrators of data, and techniques that recur in several settings. Three factors must be balanced when integrating public opinion into priority setting: first, balancing problem-solving and sense-making objectives; second, choosing between consensus-building and structured-conflict approaches; third, addressing many broad factors or a smaller set of focused alternatives. © The Policy Press • 2011 • ISSN 1744 2648.
Hide

Abstract

OBJECTIVES: This study comprises a review of public involvement strategies across the five stages of research management in the UK's HTA program at the end of a 10-year period. These five stages are: identification of topics; prioritization of these topics as researchable questions; commissioning of research; monitoring of projects throughout their implementation; and publication and dissemination of findings. METHODS: Internal HTA documentation was analyzed alongside narrative analysis of semi-structured interviews of program staff, and a rapid review of published literature. RESULTS: Public involvement strategies have developed with the growth of the HTA program but were spread unevenly across the five stages of research management. Public involvement was present in identification, strongest in prioritization, present in commissioning but minimal in monitoring and absent in publication and dissemination. CONCLUSIONS: The HTA program has developed public involvement strategies but mainly in prioritization. Further research is required to ascertain where public involvement can be most appropriately used and to evaluate its impact.
Hide

Abstract

Health care decisionmakers face increasing pressure to use health care resources more efficiently, but the information they need to assess policy options often is unavailable or not disseminated in a useful form. Findings from stakeholder meetings and a survey of private-sector health care decisionmakers in California begin to identify high-priority issues, the perceived adequacy of current information, and preferred formats and other desired attributes of research. This is a first step in establishing a systematic approach to linking the information priorities of private-sector decisionmakers with those who fund and conduct research.
Hide

Abstract

Background: Systematic approaches to involving patients and clinicians in setting agendas have been led by the Cochrane Collaboration and reported in wider research programmes for over ten years. More recently the James Lind Alliance has supported working partnerships of patients and clinicians to identify the most important gaps in knowledge about the effects of treatments. This paper scopes the research literature addressing such efforts. Methods: Electronic searches of bibliographic databases were complemented by handsearching and contacting key authors. Two researchers, initially working independently, included and described the relevant reports. Results: Over 250 studies addressed patients'or clinicians'priorities for research and outcomes for assessment. This literature described different routes for patients and clinicians to contribute to research agendas, engaging directly (148) or indirectly with research, in order to identify important areas for research, questions for research and tools for assessment. Two thirds of the studies addressing patients'or clinicians'research questions were not limited to particular conditions but were applicable more widely across health care such as nursing care or health services generally. The 27 formal studies of patient involvement revealed a literature that has grown in the last decade. Although only nine studies engaged patients and clinicians in identifying research questions together, they show that methods have advanced over time, with all of them engaging participants directly and repeatedly in facilitated debate, and most employing formal decision-making procedures. Conclusions: A sizeable literature is available to inform priorities for research and the methods for setting research agendas with patients and clinicians. This literature has yet to be appraised for the legitimacy and fairness of the participation methods or the quality of any evaluations. Nevertheless, the references are potentially useful to systematic reviewers and have been made publicly available alongside DUETS, the Database of Uncertainties of the Effects of Treatment, at www.library.nhs.uk/duets/.
Hide

Abstract

BACKGROUND: Involving members of the public in setting priorities for health research in becoming increasingly common practice. One method used in public involvement exercises is the citizens'jury. OBJECTIVE: This article examines some challenges and benefits of citizens'juries, including issues relating to process, public engagement and outcome. DESIGN: In Bristol, UK, a citizens'jury was held with the aim of identifying local priorities for research into health and social care. This jury is used as an example through which key issues in public involvement and jury processes are explored. SETTING AND PARTICIPANTS: The Bristol Citizens'Jury comprised 20 members of the public ('jurors'), an oversight panel and a steering group. The jurors met at 11 consecutive sessions during 2006 over a period of 16 weeks, which culminated in a written report. All the sessions were audio-recorded, five sessions were observed and video-recorded, and 16 jurors completed written feedback forms at the end of the jury process. FINDINGS AND CONCLUSION: In this article we discuss degree and timing of public involvement in the process of health research; the role of context; representation of communities; processes of deliberation and knowledge production; and how constraints of time and cost may affect public involvement. It was clear that jurors who took part in the Bristol Citizens'Jury were engaged and committed. This engagement may be related to jurors'belief in their ability to shape future research alongside concern about the relevance of the issues under discussion. Opposing emotions of tension and harmony are a crucial part of the deliberation process.
Hide

Abstract

OBJECTIVES: To assess the feasibility of using a citizens'jury to elicit public values on health technologies and to develop criteria for setting priorities for health technology assessment (HTA). METHODS: Sixteen individuals were selected from 1600 randomly sampled residents of the Capital Health Region in Alberta, Canada. They participated in a 2 (1/2) day jury which comprised presentations by 'expert witnesses', who represented innovators, patients, health-care policy-makers and clinicians, as well as a series of small and large group priority-setting exercises based on actual examples of technologies that had recently been considered for assessment by local and national HTA bodies. The session was audio-taped, and transcripts were independently reviewed by two researchers using content analytical techniques in order to ensure that no important concepts expressed by individual jurors were missed during group development of the final list of priority-setting criteria. Jurors evaluated the process by completing self-administered, semi-structured questionnaires at the end of the session. Responses were analysed using qualitative methods. RESULTS: The jury identified 13 criteria, which they subsequently ranked in order of importance. The top two criteria included 'potential to benefit a number of people'and 'extends life with quality'. Based on feedback from questionnaires, jurors valued the opportunity to become engaged in such a process, and expressed interest in participating in future juries. CONCLUSIONS: Citizens'juries offer a feasible approach to involving the public in priority-setting for HTA. Furthermore, technologies that may benefit a number of people and improve quality of life appear to be of greatest importance to the public.
Hide

Abstract

The discussion over whether community preferences have a legitimate role to play in priority setting has been highly polarised. Skeptics warn of the risk of establishing a 'dictatorship of the uninformed', while advocates proclaim the legitimacy of the participatory process. The one group who appears not to be consulted in this debate is the citizens themselves. In this study, a convenience sample of 373 citizens attending two medical clinics in central Sydney were surveyed about whether the general public has a legitimate role to play in informing priority setting in health care. Respondents were presented with three different levels of priority setting: across health care programmes, across medical procedures, and at a global level. To assist respondents in understanding the choices and trade-offs involved, they were given information about current levels of funding and the cost-effectiveness of each alternative. Respondents were asked whether they felt the preferences of the general public should be used to inform priority setting at each level. Of particular interest was the question of whether their willingness to use public preferences depended on the level of priority setting. Respondents were also asked about who else's preferences should be used to inform priority setting at each level. The results suggest that the public overwhelmingly want their preferences to inform priority-setting decisions in health care. This was seen to be particularly important in informing decisions about how to prioritise across broad health care programmes and about the criteria to be used to allocate funds across different population groups. In contrast, the preferences of medical professionals and health service managers were rated most highly in relation to the prioritisation of different treatments and medical procedures. In most cases, however, respondents did not advocate the use of one particular group's preferences. Even when the preferences of the general public were considered most important, it was felt that any decision-making process needed to be informed by the preferences of a range of groups. The preferences of politicians were viewed as least important to processes of priority setting in health care.
Hide

Abstract

Citizens'juries are an attempt to meaningfully involve members of the public in decisions which affect them in their own communities. The Institute for Public Policy Research and Cambridge and Huntingdon Health Authority have recently piloted the first jury in the United Kingdom. Sixteen jurors sat for four days, hearing evidence from a number of expert witnesses. The jurors were asked to consider how priorities for health care should be set, according to what criteria, and to what extent the public should be involved in this process. This pilot was also an attempt to assess the process itself, and our initial evaluation indicates that, given enough time and information, the public is willing and able to contribute to the debate about priority setting in health care.
Hide

Abstract

OBJECTIVE: We present a multistep process for identifying priority research areas in rehabilitation and long-term care of traumatic brain-injured (TBI) patients. In particular, we aimed to (1) identify which stakeholders should be involved; (2) identify what methods are appropriate; (3) examine different criteria for the generation of research priority areas; and (4) test the feasibility of linkage and exchange among researchers, decision makers, and other potential users of the research. STUDY DESIGN AND SETTING: Potential research questions were identified and developed using an initial scoping meeting and preliminary literature search, followed by a facilitated mapping workshop and an online survey. Identified research questions were then prioritized against specific criteria (clinical importance, novelty, and controversy). Existing evidence was then mapped to the high-priority questions using usual processes for search, screening, and selection. A broad range of stakeholders were then brought together at a forum to identify priority research themes for future research investment. Using clinical and research leaders, smaller targeted planning workshops prioritized specific research projects for each of the identified themes. RESULTS: Twenty-six specific questions about TBI rehabilitation were generated, 14 of which were high priority. No one method identified all high-priority questions. Methods that relied solely on the views of clinicians and researchers identified fewer high-priority questions compared with methods that used broader stakeholder engagement. Evidence maps of these high-priority questions yielded a number of evidence gaps. Priority questions and evidence maps were then used to inform a research forum, which identified 12 priority themes for future research. CONCLUSION: Our research demonstrates the value of a multistep and multimethod process involving many different types of stakeholders for prioritizing research to improve the rehabilitation outcomes of people who have suffered TBI. Enhancing stakeholder representation can be augmented using a combination of methods and a process of linkage and exchange. This process can inform decisions about prioritization of research areas. Copyright © 2013 Elsevier Inc. All rights reserved.
Hide

Abstract

Executive Summary The primary purpose of Washington Health Technology Assessment (WA HTA) Program is to ensure medical treatments and services paid for with state health care dollars are safe and proven to work. The WA HTA serves as a resource for state agencies purchasing health care by contracting with independent external vendors to produce scientific, evidence-based reports about the safety and efficacy of select medical devices, procedures, and tests. An independent clinical committee of health care practitioners uses the reports to “determine the conditions, if any, under which” programs should pay for the medical device, procedure, or test [Revised Code of Washington 70.14.080 (2006)]. Participating state agencies include the Health Care Authority; Department of Social and Health Services (Medicaid); Labor and Industries; Department of Corrections; and Department of Veterans Affairs. Little research is available on “gold standard” components of health technology assessment (HTA) programs and their processes. To assess how the WA HTA compares with the structure and processes of other HTA programs, and to assess stakeholder understanding and perception of the WA HTA , the Center for Evidence-based Policy (Center) at Oregon Health & Science University (OHSU) was engaged to conduct a “Stakeholder Engagement Project“ (the Project). The Project had two primary components: 1) a review of common components of national and international HTA programs, conducted through a collaboration of ten states (the Medicaid Evidence in Decisions project); and 2) an assessment of stakeholder perceptions of the WA HTA relating to these common components and the WA HTA program's legislative mandate. The Project was not a formal program evaluation of the WA HTA, nor did it undertake an outcome evaluation of the Program. Review of International and National HTA Programs The International Network of Agencies for Health Technology Assessment defines health technology assessment as “a multidisciplinary field of policy analysis, studying the medical, economic, social and ethical implications of development, diffusion and use of health technology” (2011). Health technology assessments conducted through formal HTA programs can be instrumental in informing public and private payer coverage and policy decisions. The scope of technologies assessed by programs varies, but commonly includes pharmaceuticals, medical devices, procedures, diagnostics, and treatment strategies (Drummond 2008). A full search of the Center's core clinical evidence sources was conducted to identify systematic reviews, meta-analyses, and technology assessments published after December 1999. Twenty-four HTA programs were identified and searched, and 12 national and international HTA programs were selected for review. Across these programs, fourteen core components were identified and organized into six broad categories: 1. HTA Organization and Structure 2. Transparency 3. Stakeholder Involvement 4. Topic Nomination and Selection 5. Evidence Synthesis 6. Use of HTA in Decision Making Overall, from the Center's review of HTA programs, the Washington HTA Program's mandates and processes are consistent with many of the core components of other well-established national and international HTA programs. Examples include: the Washington HTA Program maintains an open topic nomination process, uses an independent review committee with authority to make coverage determinations, is based on evidence synthesis, solicits public comment at common points in the topic research process, and has publically available reports, coverage decisions, and program process documents. With few exceptions, the methods employed by the WA HTA Program are similar to other publically funded HTA programs. Assessment of Stakeholder Experience Engaging stakeholders in public policy processes has been increasingly called for as a component of health services research and health care reform. As a second component of the Stakeholder Engagement Project, feedback was solicited from a full range of stakeholders regarding their knowledge and perceptions of, as well as experience with, the Program's core components and its processes. The goal was to assess stakeholder understanding and perception of the WA HTA Program's mandate and processes. A multimodal approach was used to gather feedback, allowing for a diverse range of feedback, while including opportunities for more focused discussion. Three methods were employed: an online survey, key informant interviews, and facilitated discussions. Each method builds on the findings of the one(s) preceding it. This approach is designed to assess stakeholder perception and to understand areas of concern. The process identified areas of stakeholder satisfaction and concern with the Program. However, only areas of concern were explored in-depth. Findings from all three modes of engagement efforts were similar. Stakeholders expressed support of the Program and its processes. Stakeholders understand the mandate and are supportive of an independent, decision making body. Stakeholders acknowledged and supported the HTCC's use of evidence to make decisions. They indicated that WA HTA seems interested in stakeholder perspectives and is attempting to be transparent. Processes are perceived as defined and transparent, although information, including how processes are structured, may need to be more clearly communicated and user friendly. Stakeholders discussed the need for a consistent and explicit definition of “evidence” and expressed confusion about under which circumstances which types of evidence would be “good enough.” There was also a call for clear standards and definitions, as well as for the Program to provide additional clinical and cost context for the evidence reviews. Overall Project Findings (Review of HTA Programs and Assessment of Stakeholder Experience) Since its inception in 2007, the WA HTA Program has established a solid foundation in the national and international world of evidence-based health care assessment. The Stakeholder Engagement Project found that the majority of stakeholders responding to request for feedback are satisfied the Program is successfully meeting its mandate and that current processes are fair and unbiased. The Program is at a natural point in its development to look at opportunities for continuous improvement. This Project provides valuable information that the WA HTA Program can use in ongoing improvement activities. Key findings include: § The Washington HTA Program maintains processes or generates products in that are consistent with the 14 core components of other well-established national and international HTA programs. § The purpose of the HTA is supported, its mandate is understood, and there is support for an independent, coverage decision-making body. § The Program's processes are transparent, but are not well understood, particularly by “external” stakeholders1. The Program currently publishes most of the information that stakeholders are interested in, but it is not always easy for stakeholders to find. Program areas that lack stakeholder understanding primarily include: a. Topic nomination and selection; b. HTCC member and evidence vendor selection; and c. Opportunities for public input or involvement. § Stakeholder concerns and suggested improvements are generally focused on three areas: a. Timelines for review of draft reports; b. Opportunity for public input to the HTCC; and c. Access to public comment and their disposition. There is a need to provide context for the evidence reports, including clinical background and cost information. § Tension between program timelines, evidence synthesis and decision mandates was evident from the various stakeholder voices in this Project. Recommendations Based on the findings in this report, the Center for Evidence-based Policy recommends the WA HTA consider eight modifications relevant to the areas of stakeholder communication, program processes and evidence reports. These suggestions should be considered in light of the WA HTA mandate and resources 1. Improve usability of the WA HTA website, making information of importance to stakeholders clear and available within a limited number of clicks. 2. Create simple, user-friendly diagrams and documents that summarize important Program information. 3. Conduct additional outreach to individuals and organizations, particularly patients, patient groups and providers. 4. Publish a timeline for the estimated completion of each report. 5. Review timelines for public comment of draft reports. The need for extending comment periods may be alleviated by additional public engagement in topic selection and topic refinement processes, and publication of timelines. Consider extending the comment period for draft reports. 6. Publish information regarding the disposition of public comments. 7. Clearly communicate purpose and role for each opportunity of stakeholder involvement. 8. Include contextual information by: a. Providing clinical background for the topic; and b. Providing cost data, when available, for a topic.
Hide

Abstract

Priority setting in medicine is generally regarded as an appropriate means for preparing just allocation of medical resources. By involving the general public or affected stakeholders in priority setting, advocates hope to legitimise this process and increase the acceptability of future decisions on resource allocation. Here, we differentiate between two ideal-typical methods of stakeholder involvement: 1) qualitative and 2) quantitative ones. We argue that the level of information of participants is important to the quality of the outcome of participatory events. Qualitative methods aim at fostering deliberative discussions among well-informed stakeholders. By contrast, quantitative methods usually do not have the capacity to ensure or, at least, control the level of information that participants use to guide their decisions. Hence, we conclude that in the context of priority setting qualitative and especially deliberative methods are preferable to quantitative approaches.
Hide

Abstract

Stakeholder engagement in comparative effectiveness research continues to gain national attention. While various methods are used to gather stakeholder expertise and form recommendations, evaluation of the stakeholder experience is often missing. The lack of evaluation prohibits assessing how effective and meaningful engagement practices are for enhancing research efforts and limits the ability to identify areas for future improvement. We propose that an evaluation plan of engagement processes be developed before stakeholder involvement begins and be required as part of a request for proposal or research grant where stakeholder input is being sought. Furthermore, we recommend the inclusion of six meta-criteria that represent normative goals of multiple studies: respect, trust, legitimacy, fairness, competence and accountability. To aid in the development of future evaluations, we have developed definitions for and matched specific examples of measuring each meta-criterion to serve a guide for others in the field.
Hide

Abstract

Objective. This literature review presents a summary of the state of the art in methods that organizations use to involve stakeholders – patients, consumers, practicing clinicians, payers, and others – in health care research and in activities in related fields. This research was conducted by the Community Forum, a project funded by the Agency for Healthcare Research and Quality (AHRQ), to identify emerging strategies in stakeholder engagement to enhance stakeholder involvement in the Agency's Effective Health Care Program and other health care research activities. Data sources. We reviewed peer-reviewed literature, grey literature, and Web sites, and conducted interviews with key informants (KIs) experienced in stakeholder engagement. Methods. To guide our search, we developed a conceptual framework for stakeholder engagement. We used the key word search terms listed in Appendix A to search peer-reviewed literature in academic databases and grey literature and Web sites using the Internet. We applied inclusion/ exclusion criteria (Table 1) that emphasized innovation including the use of technology. We then abstracted all sources and analyzed findings. The final review included 23 peer-reviewed articles, 15 grey literature documents, and 43 Web sites (Appendix B). Additionally, we identified 11 key informants experienced in stakeholder engagement within and outside of health care. We used a semi-structured interview protocol designed to elicit information on innovations in stakeholder engagement. The KIs provided information on many of the methods described in this report, and suggested additional organizations and Web sites to explore in the concurrent literature scan. Results. Based on the literature review, we describe types of organizations that work with stakeholders, the specific groups (eg., local residents, advocacy groups, or professional societies) that organizations identify as stakeholders, organizations'motivations for involving stakeholders, points in the research process where stakeholders can contribute, how organizations prepare stakeholders to meaningfully participate, and stakeholders'motivations for participating in research projects. We found very little that described the evaluation of stakeholder engagement processes or outcomes related to stakeholder engagement activities. A recurring theme in the literature was the importance of building trust, both for encouraging stakeholders to become involved and in maintaining their involvement. We identified a number of uses for technology in working with stakeholders throughout the research process. Many organizations use their Web pages to post available research projects and recruit for research participants or use online matching services for available projects and participants. Online collaborative platforms are used to generate ideas, promote discussion about these ideas in an online forum, and then to rank or vote on the ideas. Online communities are used to recruit participants for idea generation, to elicit feedback on product development, and to disseminate research findings. Product development challenges – contests where organizations challenge their members or the public to submit ideas for or create a product – are used to recruit new members, increase awareness of the organization issuing the challenge, and provide insight on stakeholder preferences and needs. Key informants (KIs) described examples of their stakeholders, definitions of innovation, examples of practices they have used and consider innovative, how they currently use technology, and challenges to engaging stakeholders. The KIs described a range of innovative practices, consistent with definitions of innovation that emphasized flexibility and effectiveness as well as non-traditional activities. For example a KI in health care uses blogging to translate complex material into concise and engaging articles for the organizations'stakeholders; multiple KIs mentioned using social media tools as means of eliciting input from stakeholders in prioritizing researc projects and helping to design research. Overall, the key informants highlighted the importance of measures such as selecting stakeholders that recognize the issues under discussion as a priority, communicating to stakeholders the importance of their participation and the ways in which their input will be used, ?tailoring the experience? to specific stakeholder populations, and appropriate approaches to presenting data. The group described use of technology in the form of social media marketing/advertising; social networking (Twitter, Facebook, blogs); and social media tools (IdeaScale, UserVoice, Salesforce). The KIs identified challenges to engaging stakeholders that included attracting stakeholder interest in their organization, educating stakeholders concerning their organization, incorporating new technology and methods, and resource constraints on both stakeholders and organizations. Conclusions. Based on our review, we identify five priority methods that researchers working with stakeholders may wish to consider to enhance the process of engaging stakeholders and stakeholders'ability to contribute meaningfully to the organization's activities. These methods are relevant for stakeholder recruitment and preparation and for stakeholder involvement in topic identification and prioritization, product development, and dissemination of research findings and products (Table 8). The five methods are: Online collaborative platforms: Computer software that enables interaction between an organization and its target audience through a Web site or virtual space. Collaborative platforms allow stakeholders to suggest, vote for, rank, or comment on ideas about a particular topic; they allow for frequent feedback by a forum facilitator and a feedback loop to keep stakeholders aware of how their input is being used. Product development challenges: Contests in which an organization challenges its target audience to submit ideas for or to create products. Participants compete for a chance to win prizes from the host organization, while providing input on topics of interest and generating creative ideas for dissemination and implementation. Online communities: Virtual communities where participants communicate, share ideas, and work together. Members are a subset of stakeholders that have voluntarily joined the community, making them more likely to be interested in the topic. Grassroots community organizing: Efforts using a local, ground-up approach that can be useful for spreading awareness of and building trust in an organization, for recruitment, and for product dissemination. Collaborative research: An approach to research that integrates stakeholders in stages in order to enhance the relevance of the work to the end users.
Hide

Abstract

Background/context Systematic evidence reviews (SERs) identify knowledge gaps in the literature, a logical starting place for prioritizing future research. Varied methods have been used to elicit diverse stakeholders'input in such prioritization. Objective To pilot a simple, easily replicable process for simultaneously soliciting consumer, clinician and researcher input in the identification of research priorities, based on the results of the 2009 SER on screening adults for depression in primary care. Methods We recruited 20 clinicians, clinic staff, researchers and patient advocates to participate in a half-day event in October 2009. We presented SER research methods and the results of the 2009 SER. Participants took part in focus groups, organized by profession; broad themes from these groups were then prioritized in a formal exercise. The focus group content was also subsequently analysed for specific themes. Results Focus group themes generally reacted to the evidence presented; few were articulated as research questions. Themes included the need for resources to respond to positive depression screens, the impact of depression screening on delivery systems, concerns that screening tools do not address comorbid or situational causes of depression and a perceived 'disconnect'between screening and treatment. The two highest-priority themes were the system effects of screening for depression and whether depression screening effectively leads to improved treatment. Conclusion We successfully piloted a simple, half-day, easily replicable multi-stakeholder engagement process based on the results of a recent SER. We recommend a number of potential improvements in future endeavours to replicate this process.
Hide

Abstract

Objectives. To describe methods used to engage stakeholders to prioritize future research needs from Comparative Effectiveness Reviews. Data Sources. There were three complementary phases in this project: Phase 1: Literature Scan; Phases 2 and 3: Interviews with key informants (KIs) and Evidence-based Practice Centers (EPCs) with experience engaging stakeholders. Methods. Phase 1: We searched and reviewed the literature from inception until August/September 2010 to identify methods used to engage stakeholders in identifying and prioritizing research. Phase 2: Between June 16 and July 9, 2010, KIs were interviewed about their experiences engaging stakeholders and the processes used to set research priorities. Investigators analyzed interview transcripts to identify common themes. Phase 3: The experiences of EPCs in engaging stakeholders to identify research gaps and prioritize future research needs were evaluated by: (1) review of EPC protocols and (2) semistructured interviews with EPC directors and staff. Results. Phase 1: Fifty-six studies were identified. Important considerations in stakeholder engagement included the need for consistent terminology, the intended purpose for engagement, the explicit identification of stakeholder groups, and the distribution of stakeholders. Studies frequently used a mixed-methods approach for research prioritization, combining in-person venues with quantitative prioritization processes such as voting or Delphi. Phase 2: We conducted 13 interviews. KIs used e-mail, conference calls, focus groups, and the modified Delphi technique to engage stakeholders. Processes for prioritization ranged from no formal process to a structured process that consisted of multiple rounds of voting/ranking. Phase 3: Eight EPCs were interviewed. Group conference calls were the most common approach of stakeholder engagement, along with e-mail or Web-based prioritization. EPCs routinely identified three main challenges, including timing, restriction on number of stakeholders, and limited availability of Federal stakeholders. Conclusions. Important considerations regarding stakeholder engagement to prioritize research include use of: 1. Consistent terminology and definitions throughout the process. 2. In-person methods for brainstorming, identifying topics, clarifying issues, and eliciting a deeper understanding. 3. Quantitative methods for prioritizing research.
Hide

Abstract

OBJECTIVES: This article explains how the issue of stakeholder involvement was addressed in the European network for Health Technology Assessment (EUnetHTA) Project and describes the structures of future stakeholder involvement in the EUnetHTA Collaboration. METHODS: Initiatives led to a dialogue with stakeholders and exchanging views and expectations on health technology assessment (HTA) processes and the future development of EUnetHTA. The methods of involving different stakeholder groups in EUnetHTA included general information to stakeholders about EUnetHTA, targeted information on a Web site, analysis of stakeholder opinions on HTA and EUnetHTA, and development of a draft stakeholder policy. RESULTS: First steps were taken to organize processes to consolidate the legitimacy of EUnetHTA and its products and encourage the representation of interests, thus contributing to promoting the utilization of HTA in national/regional policy making. A stakeholder Web site, analyses of stakeholder opinions on HTA and EUnetHTA in a discussion topic catalog, and a draft stakeholder policy resulted from the work. CONCLUSIONS: Stakeholder involvement in EUnetHTA is necessary to ensure the legitimacy and prospects for utilization of EUnetHTA and its products. The described activities and results create the foundation for a continued dialogue with, and involvement of, stakeholders. The EUnetHTA stakeholder meeting can be considered as a successful experience of dialogue between EUnetHTA and stakeholders, which should be continued. Our experience shows the challenge of obtaining balanced stakeholder representation across the identified stakeholder groups. Continued attention should be given to achieving balanced stakeholder representation.
Hide

Abstract

Knowledge transfer and exchange is the process of increasing the awareness and use of research evidence in policy or practice decision making by nonresearch audiences or stakeholders. One way to accomplish this end is through ongoing interaction between researchers and interested nonresearch audiences, which provides an opportunity for the two groups to learn more about one another. The purpose of this article is to describe and discuss various stakeholder engagement opportunities that we employ throughout the stages of conducting a systematic review, to increase knowledge utilization within these audiences. Systematic reviews of the literature on a particular topic can provide an unbiased overview of the state of the literature. The engagement opportunities we have identified are topic consultation, feedback meetings during the review, member of review team, and involvement in dissemination. The potential benefits of including stakeholders in the process of a systematic review include increased relevance, clarity, and awareness of systematic review findings. A further benefit is the potential for increased dissemination of the findings. Challenges that researchers face are that stakeholder interactions can be time- and resource-intensive, it can be difficult balancing stakeholder desires with scientific rigor, and stakeholders may have difficulties accepting findings with which they do not agree. Despite these challenges we have included stakeholder involvement as a permanent step in the procedure of conducting a systematic review.
Hide

Abstract

Stakeholder input is a critical component of comparative effectiveness research. To ensure that the research activities of the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Network, supported by the Agency for Healthcare Research and Quality, translate into the greatest impact for everyday practice and policy-making in cancer, we were tasked with soliciting stakeholder input regarding priority areas in cancer-related comparative effectiveness research for the DEcIDE Cancer Consortium. Given the increasing emphasis on stakeholder engagement in research, many investigators are facing a similar task, yet there is limited literature to guide such efforts, particularly in cancer care. To help fill this gap, we present our approach to operationalizing stakeholder engagement and discuss it in the context of other recent developments in the area. We describe challenges encountered in convening stakeholders from multiple vantage points to prioritize topics and strategies used to mitigate these barriers. We offer several recommendations regarding how to best solicit stakeholder input to inform comparative effectiveness research in cancer care. These recommendations can inform other initiatives currently facing the challenges of engaging stakeholders in priority setting for cancer.
Hide

Abstract

Comparative effectiveness research (CER) is intended to address the expressed needs of patients, clinicians, and other stakeholders. Representatives of 54 stakeholder groups with an interest in chronic obstructive pulmonary disease (COPD) participated in workshops convened by the COPD Outcomes-based Network for Clinical Effectiveness and Research Translation (CONCERT) over a 2-year period. Year 1 focused on chronic care and care coordination. Year 2 focused on acute care and transitions in care between healthcare settings. Discussions and provisional voting were conducted via teleconferences and e-mail exchanges before the workshop. Final prioritization votes occurred after in-person discussions at the workshop. We used a modified Delphi approach to facilitate discussions and consensus building. To more easily quantify preferences and to evaluate the internal consistency of rankings, the Analytic Hierarchy Process was incorporated in Year 2. Results of preworkshop and final workshop voting often differed, suggesting that prioritization efforts relying solely on requests for topics from stakeholder groups without in-person discussion may provide different research priorities. Research priorities varied across stakeholder groups, but generally focused on studies to evaluate different approaches to healthcare delivery (e.g., spirometry for diagnosis and treatment, integrated healthcare strategies during transitions in care) rather than head-to-head comparisons of medications. This research agenda may help to inform groups intending to respond to CER funding opportunities in COPD. The methodologies used, detailed in the online supplement, may also help to inform prioritization efforts for CER in other health conditions.
Hide

Abstract

INTRODUCTION: Professional societies, like many other organizations around the world, have recognized the need to use more rigorous processes to ensure that healthcare recommendations are informed by the best available research evidence with input from appropriate stakeholders. This is the ninth of a series of 14 articles that were prepared by an international panel to advise guideline developers in respiratory and other diseases on approaches for guideline development. We updated a review of the literature on stakeholder involvement, focusing on six key questions. METHODS: In this review we addressed the following questions. (1) What are "stakeholders"? (2) Why involve stakeholders in guidelines? (3) At what stage should stakeholders contribute to guidelines? (4) What are the potential barriers to integrating stakeholder involvement? (5) How can stakeholders be involved effectively? (6) Should anyone be excluded from the process? We searched PubMed and other databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct our own systematic reviews. Our conclusions are based on available evidence, the experience of guideline developers, and workshop discussions. RESULTS AND DISCUSSION: Stakeholders are all those who have a legitimate interest in a guideline. They include healthcare professionals, patients and caregivers, public and private funding bodies, managers, employers, and manufacturers. Their engagement is justified for several reasons, including limitations of evidence, principles of transparency and democracy, ownership, and potential policy implications. They have a role to play at different points of guideline development, but their involvement can be complex. To be successful, stakeholder engagement needs to be inclusive, equitable, and adequately resourced.
Hide

Abstract

A major priority for funding agencies and researchers involved in comparative-effectiveness research (CER) is to ensure that research questions will produce findings that are relevant and feasible to implement. In this article, we describe a process for involving experts and stakeholders in identifying and prioritizing CER studies, as illustrated by our experience in chronic obstructive pulmonary disease (COPD).
Hide

Abstract

BACKGROUND: Although patients are increasingly involved in agenda setting within specific fields of health research, they rarely participate in decision-making on the entire breadth of health research, including biomedical research. This might be attributable to a widely held view that patients are incapable of adequate research prioritization due to a lack of relevant knowledge, a failure to look beyond their own individual problems or an inability to objectively consider long-term targets. AIMS: By conducting transparent and structured consultations with asthma and chronic obstructive pulmonary disease (COPD) patients regarding their health research priorities, we aim to assess the ability of patients to prioritize research in a well-argued way. METHODS: Patients were consulted through seven focus groups, a feedback meeting, and a questionnaire. The focus groups and the feedback meeting aimed to explore the entire breadth of patients'problems experienced in relation to their diseases, while the questionnaire aimed to investigate patients'prioritization of possible research targets focused on solving these problems. RESULTS: The focus groups produced a wide range of problems, including those related to health-care organization, social environment, therapy and costs. In terms of research prioritization, patients focused primarily on biomedical issues, particularly aetiology, co-morbidity and effective medication. CONCLUSIONS: The consultation procedure successfully elicited patients'research priorities including the underlying arguments. Our results indicate that asthma and COPD patients are capable of research prioritization in a well-argued way and that they highly value biomedical research. Furthermore, as they prioritized some research topics that are not covered by current Dutch research programmes, we argue that patient participation can broaden research agenda setting.
Hide